Applying Feedback Control to Stream Processing Systems

Processing log data from a large distributed system is challenging. The collection and analyzing such data requires timely and accurate pre-processing and sanitizing the data, which is best done with a stream processing system. This paper explores ways to add control theory to a streaming log processing system to improve its accuracy and throughput in the face of varying loads and disturbances. We introduce a way to regulate the internal queue occupancy, preventing data loss. We also provided an accurate and reliable data source to feed the analysis system using feedback control. We compare the results of two different types of commonly used controllers, Proportional Control (P Control) and Proportional-Integral Control (PI Control). We discussed our experience in different stages of applying control theory to a practical system and showed the benefits of designing a system on a solid theoretical foundation.